WhaleWatch: An intelligent model-based mathematics tutoring system

Mathematics training is essential for participation in science and engineering careers. Yet many students, especially girls, dislike and avoid math, and are therefore underprepared for university science majors and graduate programs. The goal of this project is to increase students’ interest in math and their confidence in their ability to learn math through an intelligent, model-based multimedia computer tutor, WhaleWatch, designed to teach fractions concepts. Based on a dynamically updated student model, WhaleWatch selects problems of appropriate difficulty and provides help and instruction as needed. The results of an evaluation study with 12-year-old students indicated that WhaleWatch had a positive impact on students’ math self concept and beliefs in the value of learning mathematics. The benefits were especially apparent for girls, and for students who reached the most difficult problems. WhaleWatch: An intelligent model-based mathematics tutoring system Mathematics is a critical prerequisite for many college majors and science careers. Yet in the transition from elementary school to middle school, when mathematics concepts become more abstract, many students begin to lose confidence in their ability to do well in math. Thus, when mathematics courses become optional at the secondary level, many students avoid taking math courses beyond basic algebra and geometry. They are subsequently unprepared for the many college majors and graduate programs in both the natural and social sciences that require mathematics and statistical training. In the United States and in many European countries, math avoidance is especially prevalent among female students, who subsequently perform less well than their male classmates on math achievement tests (Beal, 1994; Beller & Gafni, 1996). Even the most mathematically gifted girls have less math training and are less interested in science careers than their male peers (Benbow, 1992). When compared with boys, girls have an unrealistic pessimism about their math abilities; that is, they come to feel that they lack ability while actually performing well. In contrast, boys tend to remain quite optimistic about their abilities through college. Girls are especially vulnerable in the area of math, as compared to other academic subjects, because they typically believe that native ability or talent is especially important for learning math. As a result, girls are more easily discouraged about their performance in math when failure occurs or when they encounter the new and challenging math material that is presented in late elementary school (e.g., fractions, geometry). The gender differences in mathematics interest and self concept can also be linked to students’ experiences in math classes, in which girls receive less overall instruction, and less effective instruction. Specifically, teachers provide girls with less detailed information about how to solve problems, and how to correct errors. Some researchers have found that boys receive up to eight times more instructional feedback from mathematics teachers than girls (Boggiano & Barrett, 1991). Teachers tend to assume that girls have already done the best they can, and so they are somewhat reluctant to push a female student who has made errors. In contrast, they typically assume that boys have simply not paid attention or tried hard enough, and that boys can succeed with additional help and effort. In addition, teachers -the majority of whom are female -themselves have relatively little training in teaching mathematics and report that math is their least favorite subject to teach. Thus, teachers themselves do not provide strong role models for mathematics proficiency, nor to they convey convincingly to students that math is important and valuable to learn. The traditional form of math instruction does not convey to students how math can be used in other areas of the curriculum and in real-life tasks. Thus, it is not surprising that girls conclude that math is not worth the effort to master. By the end of elementary school, girls typically report enjoying other academic subjects, such as writing and reading, more than math, and rate math as less valuable to learn than do boys. One potential solution to the problem of math avoidance is to use intelligent model-based tutoring systems to provide more effective instruction. Such tutoring systems have been shown to be effective in maintaining student interest and confidence, and they are a highly efficient form of instruction for all students, across age, expertise, and gender (Anderson, Corbett, Koedinger, & Pelletier, 1995). We hypothesized that this form of instruction would be especially helpful to girls in the area of mathematics, because problems can be selected on the basis of the student model that are appropriately challenging, while at the same time, help and instruction are immediately available if the student has difficulty solving a problem. Intelligent model-based tutoring is also appealing to students. It is free of the unconscious stereotypes and lower expectations that can influence teacher behavior towards female students. In addition, students report appreciating that they can receive immediate help from the tutor rather than the teacher, and that interactions with the teacher become less embarassing as a result (Schofield, EurichFulcer, & Britt, 1994). The WhaleWatch mathematics tutor In the present project, we developed a model-based intelligent tutor called WhaleWatch to teach fractions concepts. The domain of fractions was selected for three reasons: First, it represents an introduction to more abstract math material and is often regarded as considerably more difficult by students than arithmetic. Second, fractions is taught to 12-year-olds in the United States, which is the point at which students, particularly girls, begin to express a dislike of mathematics. Third, there are well documented and systematic error patterns associated with specific types of fractions problems, which can be diagnosed easily by the student model. In contrast to much existing male-oriented educational software, WhaleWatch was specifically designed to support the learning styles that are appealing to girls. Problems involving fractions are presented in the context of the domain of environmental biology which, of the sciences, is particularly appealing to girls. In the present version of the system, problems focus on a local endangered species, the Right Whale (hence the name, WhaleWatch). When the student begins the program, he or she reads several screens containing introductory descriptive information about the animals, including graphic images, and current data about the species’ status. The student then takes on the role of an environmental biologist who is charged with monitoring the status of the species. Whole number and fractions problems are presented that involve questions about the animals’ migration; reproduction; changes in population over time; and impact of changing environmental conditions. The value of learning math is conveyed through the overall goal of assisting the animals, rather than through competition, which girls tend to dislike as the basis for a learning activity. WhaleWatch also provides individualized instruction and supportive feedback to each student. For example, if a student fails twice to enter the correct answer to an addition-of-fractions problem, he or she is asked "would you like to see an example?" If the student clicks "yes," a Quicktime movie is shown on the screen in which the solution algorithm for a problem is actively and graphically displayed. If the student model infers that the student does not understand a key concept (e.g., that the denominators of fractions must be the same for them to be added), an instructional screen appears that provides relevant information, and if the student continues to make errors, he or she is walked through the problem solution with increasingly detailed guidance. This individualized instruction and help is based on the program’s student model feature, a representation of what the student understands about the math concepts that is continually updated and revised. The student model ensures that students do not become overly discouraged; they learn that difficult problems can be solved with practice and persistance, and that they are capable of learning the material. We hypothesized that this feature should be particularly effective with female students, who are more easily discouraged about their progress in math. A more detailed description of the student model used in WhaleWatch is presented next. Representing the student model in WhaleWatch The student model is used to record information specific to each individual student. Two types of knowledge are gathered about the student: his or her scores on individual topics in the domain being taught, and general factors that are applicable to the whole domain. Individual topic proficiencies The student model in WhaleWatch records a proficiency for each domain topic. We are not specifically interested in low-level representation issues associated with student models, and are therefore not concerned if the system’s beliefs are generated using methods such as Bayesian networks or the Dempster-Shafer theory of evidence. However, we are interested in adding additional knowledge to the student model that has traditionally been neglected. Therefore a framework that can be quickly modified to permit easy experimentation is needed. For these reasons, we use a simple scheme for representing uncertainty on each topic belief vectors. This representation is based on the “fuzzy” distributions first used in Katz, Lesgold, Eggan and Gordin (1993) and Gurer, DesJardins, and Schlager (1995). Each vector contains seven points, with the values summing to 1. The value at each point indicates the approximate probability that the student is at that level of knowledge. The lowest value for the vector is (1 0 0 0 0 0 0) and the highest value is (0 0 0 0 0 0 1). A vector of (0.14 0.28 0.4 0.18 0 0 0) means the tutor believes there is approximately a 14% chance the student is at level 1, a 28% chance she is at level 2, a 40% chance she is at level 3, and an 18% that she is at level 4. There is no chance that she is at levels 5, 6, or 7. General factors In addition to the student model storing information on each topic within the domain, we also maintain general factors concerning the student. Prior research with the LISP tutor (Anderson, 1993) and with Stat Lady (Shute, 1995) indicates that general factors such as acquisition and retention are predictive of overall learning, and allow for a more accurate response to the idiosyncrasies of the student. However, neither of these systems used this additional knowledge to reason about the student. Our system considers both acquisition and retention factors when making tutoring decisions. Acquisition records how well students learn new topics. When a new topic is introduced, the tutor views how the student performs on the first few problems. If the student performs well, then she is learning skills quickly, and her acquisition factor will reflect this. However, if a student requires many problems on a given topic before she illustrates that she understands it, then her acquisition will be lower. Retention measures how well a student remembers the material over time. This factor is updated when the student is presented with a problem on a topic that she has not seen for a given period of time. If she answers the problem correctly without requiring any hints, then she has retained the knowledge, so her retention factor will be high. On the other hand, if she needed many hints to answer the problem, and previously she did not, then her retention is poor. Adaptive feedback When a student makes a mistake while solving a problem, the tutor provides a hint to help her to continue to solve the problem. The goal is to provide the minimum amount of support needed for the student to solve the problem. Therefore, if a student is just learning a new topic and makes a mistake, the system will provide detailed instructions on how to correctly perform the incorrect step. A student who is more proficient and makes a mistake will be given a subtler prompt such as, “Are you sure about your answer?” The system analyzes the student's answer to determine the specific area in which the student needs assistance. This is determined by the expert module, which is similar to production rules. Each step in the problem solving sequence can test if the student's actions were performed correctly. Therefore the system can analyze a student's response and determine which step was probably performed incorrectly. The system now knows with which topic the student needs help, but must still determine how much support is required. To do this, the tutor checks the student model and finds the student's proficiency on this topic. If the student has a high proficiency she will be given a subtle hint; a low proficiency will result in a more explicit hint. If the student continues to make mistakes, more specific hints will be successively generated. In order to do this it is necessary to give each hint a level. This level rating runs from 1 to 7, where 1 is an explicit hint and 7 is a general hint. Updating the student model When the student finishes solving a problem, WhaleWatch examines how many (if any) hints were required. WhaleWatch then updates the student model according to what type of hints the student required for each skill necessary to solve the problem. We have extended Gurer's “fuzzy vector” representation by changing the update rules to account for the general factor described above. The original update rules are as follows: Upgrade rule: Pi = Pi – cPi + cPi-1 Downgrade rule: Pi = Pi– cPi + cPi+1 c is a constant that controls the rate of updating, and pi is the value of slot i in the belief vector. The upgrade rule is used when a student answers a problem correctly, the downgrade rule whenever a student answers incorrectly. These rules have the effect of shifting the distribution of numbers to either the left (lower proficiency) or to the right (higher proficiency). We extend this by replacing the c constant in the upgrade rule with a variable that is based on the student's acquisition factor (because this describes how quickly a student learns). Similarly we replace the c in the downgrade rule with a number that is determined by the student’s retention. This alters how quickly the model believes the student is learning/forgetting based on actual data about this student, rather than the same constant for all students. Additionally, rather than shifting the entire belief vector to the left or the right the system moves its belief towards a value corresponding to the student’s level of knowledge. It does this by using the hint level the student required to solve the problem as the target value. Items to the left of this in the vector are increased, items to the right of this are downgraded. For example, if the student required a level 6 hint (a fairly subtle prompt), the seventh slot in the vector would be given the modified downgrade rule, and the first five slots would be given the upgrade rule. As a result of the two changes described, the update rule takes the form: For i less than the hint level: Pi = Pi – APi + APi-1 For i greater than the hint level: Pi = Pi– BPi + BPi+1 where A is a function of the acquisition factor, and B is a function of the retention factor. Problem Generation The student model is used to dynamically create problems that are appropriately difficult for each student. Little prior work has been done in this area (Beck, Stern, & Woolf, 1997; Carberry & Clarke, 1997). However, clear benefits exist for constructing items “on the fly” while the program is being run rather than enumerating the possibilities ahead of time. First, dynamic generation is less time-expensive in that a model is created which can generate a wide variety of problems. Second, a broader variety of problems can be dynamically constructed by the system, as it is unlikely for the systems' authors to consider all contingencies ahead of time when building the problem database. These benefits apply to areas other than problem generation, and hold true for other systems that wish to have dynamic actions rather than static ones. To construct a “good” problem, the system must balance constructing problems which cause the student to practice skills in which she is less proficient, and constructing problems that are not too difficult for her. The system examines the student's proficiency at solving problems of the type it is preparing. For example, if the system is building an “add fractions” problem it evaluates the student's proficiency in this skill. The system controls problem difficulty by adjusting the number of subskills the student must apply to solve a problem. A topic's subskills are smaller steps that may have to be performed to solve a problem. For example, the subskills of adding fractions would be finding a least common multiple, converting both fractions to have equivalent denominators, simplifying the resulting answer, and making the fraction proper. A hard add fraction problem would be 2/3 + 3/4 common multiple and convert the operands to have equivalent denominators. An easier problem would require a student to apply none of these skills, for example 1/3 + 1/3, which simply requires the student to add the numerators. WhaleWatch selects a problem of the appropriate level of difficulty by generating problems randomly, and selecting a problem that requires the appropriate number of subskills to be performed. The number of subskills that must be performed is determined by the student’s overall proficiency. A higher proficiency indicates more subskills will be required while a lower proficiency indicates fewer subskills will be required to solve a problem. Additionally, the system gives more weight to subskills in which the student is less proficient. This results in the student’s efforts being concentrated in areas in which she needs more practice. Because the problem difficulty is also controlled, even though a beginning student needs practice on all facets of a skill, she isn’t overwhelmed with problems that are too complex. For more details of this process see Beck et al. (1997). In the next section, we describe the findings of an evaluation study conducted with WhaleWatch. Evaluation Study The participants included 50 students attending two sixth grade (ages 11-12 years) classes at an elementary school located near the University. Most of the students were white, and were generally from affluent home backgrounds. The school was equipped with a computer laboratory that the students used regularly, so they were already familiar with general computer operations. Because we believed that WhaleWatch would be beneficial to all students, both boys and girls were included in the study. To assess the students’ changing beliefs about their math ability, a questionnaire were administered in a pre and post test design. The questionnaire was drawn from prior work by Eccles, Wigfield, Harold, and Blumenfeld (1993). The questionnaire includes items that tap two dimensions of beliefs about math. Items such as "Are you among the best or worst in math in your class" assess students’ math self concept. Items such as "For you, how useful is learning math?" evaluate the students’ beliefs that math is important and valuable to learn. Students rate their response to each item on a seven point scale. The questionnaire has been shown to be highly reliable and to have good psychometric properties, and it has been frequently used in prior research on mathematics achievement. The pretest questionnaire was presented on screen when students logged in for their first Whalewatch session. After completing it, the students then worked with Whalewatch on a daily basis for a total of 6 hours over the course of one week. Researchers were available in the computer laboratory to provide assistance, along with the school computer specialist. Classroom teachers observed but were not actively involved in the sessions. At the final session, students completed the post test questionnaire on-line in an exit screen. Questionnaire responses and data regarding student performance in problem solving were automatically collected by WhaleWatch for subsequent analysis. As may be seen below, at pretest the boys had higher scores than girls for both math self confidence and beliefs in the value of learning math. However, at post test the girls’ ratings had increased significantly. This result indicates that working with Whalewatch had a positive effect on girls’ attitudes about mathematics; in fact, at post test there were no significant gender differences for either math self confidence or math value. Change in math self-confidence S u rv e y r a ti n g